Head-up display

ABSTRACT

A main controller of an HUD is configured to: acquire road surface information of a front road surface based on a plurality of measurement points on the front road surface, which is detected by a road surface detection sensor mounted on a vehicle with the HUD and is positioned forward of a traveling direction of the vehicle; calculate, by using the road surface information, a virtual image plane position on a virtual image plane where a virtual image of a display object is displayed, which is a display position of the virtual image for displaying the virtual image along the front road surface; and calculate a display position of a display surface within an image display device, which corresponds to the virtual image plane position, so as to output, to the image display device, a control signal for displaying the display object on the display position of the display surface.

TECHNICAL FIELD

The present invention relates to a head-up display (Head-Up Display:HUD).

BACKGROUND ART

Patent Literature 1 discloses an automotive display system which“includes an image projection unit and an angle information acquisitionunit. The image projection unit projects a light flux including an imagewith a display object toward one eye of an image viewer. The angleinformation acquisition unit acquires at least one of vehicle angleinformation and external environment angle information. The vehicleangle information relates to an angle of at least one of attitudes and aheading of a vehicle carrying the image viewer. The external environmentangle information relates to an angle of a background object at a targetposition of the display object in a background of an externalenvironment of the vehicle. The image projection unit changes an angleof the display object in the image based on at least one of the vehicleangle information and the external environment angle informationacquired by the angle information acquisition unit” (excerpted fromAbstract).

CITATION LIST Patent Literature

-   Patent Literature: JP-A-2010-156608

SUMMARY OF INVENTION Technical Problem

In sensing performed by a general monocular camera, a distance isestimated by referring to a point (vanishing point) at which parallelwhite lines drawn on the road appear to intersect with each other andcalculating the distance based on the height (angle) from the vanishingpoint. Since the calculation above assumes that all in the field of viewhave the same height, if applying the height calculation methoddescribed above to a non-flat road surface with slope on its front, notonly the distance from the vanishing point, but also the height of theroad surface due to the slope is included in the calculation result. Onthe coordinates of the road surface with the slope, the height of theroad surface due to the slope is added to the height calculated based onthe vanishing point, and accordingly, an error that is not small isincluded in the coordinates of the road surface with the slope. In orderto perform AR display in accordance with the shape of the front roadsurface, the three-dimensional coordinates of the road surface arenecessary. However, in the case of using the monocular camera solely,only the flat road surface allows accurate calculation. Therefore, thereremains a problem that the AR display along the slope of the roadsurface cannot be performed.

The present invention has been made in view of the circumstancedescribed above, and an object of the present invention is to provide anHUD capable of performing AR display for a road surface with slope moresuitably.

Solution to Problem

In order to solve the problem above, the present invention includes thetechnical features described in the scope of claims. As one aspect ofthe present invention, it is provided a head-up display for irradiatingan image light including a display object toward a projection targetmember so as to display the display object as a virtual image, thehead-up display comprising: an image display device including a lightsource and a display surface, the image display device being configuredto output the image light generated by a light which has been emittedfrom the light source and transmitted through the display objectdisplayed on the display surface; a virtual image optical systemconfigured to enlarge and project the image light; and a main controllerconnected to the image display device, wherein the main controller isconfigured to: acquire road surface information of a front road surfacebased on a plurality of measurement points on the front road surface,the plurality of measurement points being detected by a road surfacedetection sensor mounted on a vehicle with the head-up display and beingpositioned forward of a traveling direction of the vehicle; calculate,by using the road surface information, a virtual image plane position ona virtual image plane where the virtual image of the display object isdisplayed, the virtual image plane position being a display position ofthe virtual image for displaying the virtual image along the front roadsurface; and calculate a display position of the display surfacecorresponding to the virtual image plane position so as to output, tothe image display device, a control signal for displaying the displayobject on the display position of the display surface.

Advantageous Effects of Invention

According to the present invention, it is possible to provide an HUDcapable of performing AR display for a road surface with slope moresuitably. The problems, configurations, and effects other than thosedescribed above will be clarified by explanation of the embodimentsbelow.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic configuration diagram of an HUD.

FIG. 2 is a system configuration diagram of an HUD.

FIG. 3 illustrates a flow chart of a flow of virtual image displayprocessing by an HUD.

FIG. 4A illustrates an outline of estimation processing for a front roadsurface (y-z plane).

FIG. 4B illustrates an outline of estimation processing for a front roadsurface (x-y plane).

FIG. 5 illustrates a three-dimensional position in a real space in whicha display object is to be drawn (y-z plane).

FIG. 6A illustrates a shape of a display object parallel to a groundsurface (left in FIG. 6A; basic shape) and shape of the display objectobtained by rotating the basic shape thereof in accordance with a roadsurface angle (right in FIG. 6A; rotation shape).

FIG. 6B illustrates a state where a display object is displayed along afront road surface (y-z plane).

FIG. 7 illustrates a position on a virtual image plane for displaying avirtual image (virtual image plane position P′″_(oi)) in order todisplay the virtual image of a display object after rotation along afront road surface (y-z plane).

FIG. 8 illustrates positional relationship between a virtual image planeposition and a position on a display surface of a display element.

FIG. 9 illustrates an example in which AR display is performed for afront road surface without change in slope.

FIG. 10 illustrates an example in which AR display is performed for afront road surface having change in slope, without considering theslope.

FIG. 11 illustrates an example in which AR display is performed for afront road surface having change in slope, with considering the slope.

FIG. 12A illustrates a state where an obstacle is viewed in a LiDARsystem.

FIG. 12B illustrates a state where the same obstacle as that in FIG. 12Ais viewed in a camera coordinate system.

FIG. 13A illustrates processing for conversion from a road surfaceposition on camera coordinates into a position on a camera projectionplane.

FIG. 13B illustrates processing for conversion from a position on acamera projection plane into a position on an image.

FIG. 14 illustrates a virtual image plane position of an obstacle (y-zplane).

FIG. 15A illustrates an example in which AR display is performed for anobstacle without considering slope of a front road surface.

FIG. 15B illustrates an example in which AR display is performed for anobstacle with considering slope of a front road surface.

FIG. 16 illustrates processing for obtaining height of a display objectin a real space according to a third embodiment.

FIG. 17 is a system configuration diagram of an HUD according to afourth embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to the drawings. Throughout the drawings forexplaining the embodiments, the same members are provided with the samereference signs in general, and repetitive explanation thereof will beomitted. In each of the embodiments described below, an example in whicha head-up display (HUD) is mounted in an automobile as a vehicle will bedescribed, meanwhile, the vehicle may be a train or a working machinesuch as a hydraulic excavator. Furthermore, since the example of anautomobile as a vehicle will be described in the present embodiment,tires thereof correspond to a traveling body. Meanwhile, in the case ofa train, wheels correspond to the traveling body, and in the case of aworking machine, a crawler corresponds thereto.

First Embodiment

With reference to FIG. 1 and FIG. 2, a configuration of an HUD 1according to the present embodiment will be described. FIG. 1 is aschematic configuration diagram of the HUD 1. FIG. 2 is a systemconfiguration diagram of the HUD 1.

As illustrated in FIG. 1, the HUD 1 is provided in a dashboard 4 of avehicle 2. The dashboard 4 includes a dashboard opening 7 which allowsan image light L emitted from the HUD 1 to pass therethrough. The imagelight L is reflected by a wind shield 3 of the vehicle 2 and madeincident on the eye of a driver 5. The driver 5 visually recognizes avirtual image 101 of an arrow display object by the image light Lfurther forward than the wind shield 3. A projection target member isnot limited to the wind shield 3, and other members such as a combinermay be used as long as it is a member onto which the image light L isprojected.

The HUD 1 includes an outer housing 50, an HUD controller 20 and animage display device 30 to be mounted on the outer housing 50, and avirtual image optical system 40 for enlarging and projecting the imagelight L emitted from the image display device 30.

On an upper surface of the outer housing 50, a housing opening 51serving as an exit port of the image light L is formed. The housingopening 51 is covered with an antiglare plate 52 for preventing dust andthe like from entering the outer housing 50. The antiglare plate 52 isformed by a member that transmits a visible light.

The image display device 30 is configured by using an LCD (LiquidCrystal Display). More specifically, the image display device 30includes a light source 31, an illumination optical system 32, and adisplay element 33 that emits the image light L including a displayobject (see FIG. 2). The illumination optical system 32 is disposedbetween the light source 31 and the display element 33, and configuredto guide a light emitted from the light source 31 to the display element33.

The virtual image optical system 40 is configured by a lens unit 43 anda concave mirror 41 which are arranged in the order of proximity of theimage display device 30 along an emission direction of the image lightL. Furthermore, the virtual image optical system 40 according to thepresent invention includes a concave mirror drive unit 42 configured torotate the concave mirror 41. Although not illustrated in FIG. 1, afolding mirror that folds back an optical path of the image light L maybe provided between the lens unit 43 and the concave mirror 41. Thefolding mirror is a mirror that reflects the image light L emitted fromthe lens unit 43 toward the concave mirror 41. When providing thefolding mirror, the optical path of the image light L can be furtherlengthened, which makes it possible to display a virtual image plane 100further forward. In FIG. 1, the virtual image 101 of the arrow displayobject which represents left turn is displayed as a display object onthe virtual image plane 100.

The lens unit 43 is an assembly of at least one or more lenses foradjusting an optical distance between the concave mirror 41 and theimage display device 30.

The concave mirror 41 is a member that reflects the image light L whichhas been transmitted through the lens unit 43 toward the housing opening51. The concave mirror 41 is rotated by the concave mirror drive unit42. The concave mirror drive unit 42 is configured, for example, by amirror rotation axis and a motor that rotates the mirror rotation axis.When rotation of the motor is transmitted to the mirror rotation axis ofthe concave mirror 41, the concave mirror 41 is rotated, whereby theimage light L is reflected toward the wind shield 3 with a reflectionangle thereof being changed. When a projection direction of the imagelight L is changed, the reflection angle of the image light L on thewind shield 3 is changed. In this way, when the reflection angle of theimage light L is changed, the height of the virtual image plane 100 (seeFIG. 7) itself is changed.

The HUD 1 according to the present embodiment includes the technicalfeature in which a display object is displayed as a virtual image by AR(Augmented Reality) along slope of a front road surface 210 (see FIG.4A) positioned forward in a traveling direction of the vehicle 2.Accordingly, the HUD 1 is configured to adjust the height (height in thereal space, hereinafter referred to as “display object height”) from aground surface 200 of front wheels 6 and the rear wheels at the time ofdisplaying the virtual image 101 on the virtual image plane 100 (seeFIG. 7). In the present embodiment, adjustment of the display objectheight is realized by changing a display position of the virtual image101 on the virtual image plane 100 (hereinafter referred to as “virtualimage plane position”). In this connection, when the height of thevirtual image plane 100 itself is changed by changing an angle of theconcave mirror 41, in the case of displaying the virtual image 101 atthe same virtual image plane position on the virtual image plane 100 ofwhich the height has been changed, the display object height is shiftedby the height difference of the virtual image plane 100 itself. In thefollowing, for convenience of explanation, it is assumed that thevirtual image plane 100 itself is not changed, that is, the angle of theconcave mirror 41 is not changed.

On a front surface of the vehicle 2, a LiDAR (Light Detection andRanging) 60 as a road surface detection sensor is installed. Aninstallation position and height of the LiDAR 60 illustrated in FIG. 1is merely an example, and its installation position and height may bedifferent from the illustrated example.

On an upper portion of the wind shield 3 inside the vehicle 2, a camera70 as an obstacle detection sensor is installed, and on the dashboard 4,a GPS (global positioning system) receiver 80 as a position calculationsensor is installed. An installation position of the camera 70illustrated in FIG. 1 is merely an example, and it may be installedoutside the vehicle 2. An installation position of the GPS receiver 80is also merely an example, and the installation position thereof is notlimited on the dashboard 4.

As illustrated in FIG. 2, in the vehicle 2, an automatic operationsystem 900 and the HUD 1 are connected via an onboard network (CAN:Control Area Network) 90.

The automatic operation system 900 mainly includes a travel drive device400, a travel controller 500, a road surface estimation device 600, animage processing device 700, and a navigation device 800. The traveldrive device 400 includes an engine controller 410, a steering motor420, and a brake 430. The travel controller 500 acquires road surfaceinformation, obstacle information, map information, and navigationinformation from the road surface estimation device 600, the imageprocessing device 700, and the navigation device 800 via the CAN 90,respectively, and uses the information to output control signals forperforming an automatic operation, such as an engine control signal, asteering angle signal, and a brake signal, to the travel drive device400.

The HUD 1 may be configured to perform processing of calculating anestimation formula which expresses the front road surface 210 (roadsurface estimation processing), meanwhile, in the present embodiment, anexample in which the road surface estimation device 600 serving as oneof the elements constituting the automatic operation system 900calculates the estimation formula of the plane of the front road surface210 based on measurement point data from the LiDAR 60 so as to outputthe road surface information including the above to the HUD 1 will bedescribed. Furthermore, in the present embodiment, it is assumed thateach of the camera 70, the image processing device 700, the GPS receiver80, and the navigation device 800 serves as one of the elementsconstituting the automatic operation system 900 and is used forprocessing of displaying the virtual image 101 of the display object inthe HUD 1. Similarly, to the case of the road surface estimation device600, the camera 70, the GPS receiver 80, etc. may be configured asdedicated products of the HUD 1.

In the first embodiment, the LiDAR 60 measures a distance and a positionto a plurality of measurement points on the front road surface 210 (forexample, measurement points P₁, P₂, P₃ in FIG. 4A) to generate themeasurement point data, and the road surface estimation device 600calculates the estimation formula in which the plane including themeasurement point data is defined by a three-axis rectangular coordinatesystem in the real space. Then, the road surface estimation device 600outputs the estimation formula to the HUD 1 as road surface information.Furthermore, the road surface estimation device 600 may be configured tocalculate an angle (road surface angle θ_(r): see FIG. 4A) formed by theplane of the front road surface 210 with respect to the ground surface200, and output the calculated angle to the HUD 1 as the road surfaceinformation as well. Meanwhile, the HUD controller 20 may be configuredto calculate the road surface angle θ_(r). In a third embodiment whichwill be described later, the road surface information only includes theroad surface angle θ_(r).

The “three-axis rectangular coordinate system in the real space” isdefined by a two-axis rectangular coordinate system included in theground surface 200 (x-z coordinate system) and the y-axis perpendicularto the two-axis rectangular coordinate system. The x-axis corresponds tothe lateral axis of the vehicle 2, z-axis corresponds to thelongitudinal axis along the traveling direction of the vehicle 2, andthe y-axis corresponds to the height direction axis from the groundsurface 200.

The image processing device 700 detects an obstacle located in front ofthe vehicle 2 based on the output (captured image) from the camera 70,and the HUD 1 acquires virtual image display target object informationindicating a type and a position of the obstacle.

The navigation device 800 calculates a current position of the vehicle 2based on the output from the GPS receiver 80 (GPS radio waves), and theHUD 1 acquires positional information of the vehicle 2.

The HUD controller 20 includes a first ECU (Electric Control Unit) 21, afirst nonvolatile memory (ROM) 22, a memory (RAM) 23, a light sourceadjustment unit 24, a distortion correction unit 25, a display elementcontrol unit 26, a first CAN communication unit 27, and a concave mirrorcontrol unit 28. The first CAN communication unit 27 is connected to theroad surface estimation device 600, the image processing device 700, andthe navigation device 800, respectively, via the CAN 90. The lightsource adjustment unit 24 is connected to the light source 31, thedistortion correction unit 25 is connected to the display elementcontrol unit 26, and the display element control unit 26 is connected tothe display element 33. The concave mirror control unit 28 is connectedto the concave mirror drive unit 42.

The road surface estimation device 600 is configured by connecting asecond ECU 601, a second CAN communication unit 602, and a LiDAR controlunit 603. An input stage of the second ECU 601 is connected to an outputstage of the LiDAR 60, and the output of the LiDAR 60 (measurement pointdata) is input to the second ECU 601.

Each measurement point data includes a value obtained by calculating adistance and a position to the measurement point based on lightintensity and a laser flight time of a reflected wave received from apoint (measurement point) of the front road surface 210 where a laserlight, which is irradiated toward the front road surface 210 by theLiDAR 60, hits.

The second ECU 601 calculates the estimation formula for a planeincluding three or more pieces of the measurement point data and theangle (road surface angle θ_(r)) formed by the plane with respect to theground surface 200, and transmits the calculation result as the roadsurface information through the second CAN communication unit 602 to theHUD controller 20.

An output stage of the second ECU 601 is connected to an input stage ofthe LiDAR 60 via the LiDAR control unit 603. The second ECU 601 outputsa control signal to the LiDAR 60 via the LiDAR control unit 603.

The image processing device 700 is configured by connecting a third ECU701, a third CAN communication unit 702, and a camera control unit 703.An input stage of the third ECU 701 is connected to an output stage ofthe camera 70, and the captured image generated by the camera 70 isinput to the third ECU 701. The third ECU 701 performs image recognitionprocessing on the captured image to determine whether a subject of thecaptured image is the virtual image display target object, for example,a course display object or an obstacle. When the subject is the virtualimage display target object, the third ECU 701 transmits the virtualimage display target object information indicating the type and theposition thereof to the HUD controller 20 through the third CANcommunication unit 702.

An output stage of the third ECU 701 is connected to an input stage ofthe camera 70 via the camera control unit 703. The third ECU 701 outputsa control signal to the camera 70 via the camera control unit 703.

The navigation device 800 is configured by connecting a fourth ECU 801,a fourth CAN communication unit 802, and a fourth nonvolatile memory803. An input stage of the fourth ECU 801 is connected to the GPSreceiver 80, and the fourth ECU 801 calculates a current position of thevehicle 2 based on the GPS radio waves received from the GPS receiver 80and transmits the positional information to the HUD controller 20through the fourth CAN communication unit 802. The fourth ECU 801 may beconfigured to calculate a route to a destination of the vehicle 2 andtransmit route information to the HUD controller 20. In the following,the positional information and the route information are collectivelyreferred to as course information.

An output stage of the fourth ECU 801 is also connected to the fourthnonvolatile memory 803, and the position signal is accumulated thereinalong the time series. The fourth ECU 801 may be configured to read outthe previous positional information and obtain the time-series change ofthe positional information so as to calculate the traveling direction ofthe vehicle 2. Furthermore, the fourth ECU 801 may be configured toexecute dead reckoning processing based on the previous positionalinformation, correct the current position obtained from the GPS radiowaves by using the result, and output the corrected current position tothe HUD controller 20. The fourth nonvolatile memory 803 may store mapinformation.

The travel controller 500 includes a fifth ECU 501 and a fifth CANcommunication unit 502. The fifth ECU 501 acquires the road surfaceinformation, the virtual image object information, and the courseinformation from the fifth CAN communication unit 502 via the CAN 90,and outputs control signals to the travel drive device 400. In thisconnection, an alarm 450 may be connected to the travel controller 500.The fifth ECU 501 executes collision possibility determinationprocessing by using the virtual image display target object information,and if there is a risk of collision, outputs an alarm signal to thealarm 450. In this case, alarm notification by the alarm 450 may besynchronized with virtual image display for an obstacle of the HUD 1,which will be described later. Each of the light source adjustment unit24, the distortion correction unit 25, the display element control unit26, the concave mirror control unit 28, the LiDAR control unit 603, thecamera control unit 703, and a scanning mirror control unit 26 a used ina fourth embodiment which will be described later, may be configured bycooperating an arithmetic element such as a CPU or an MPU with a programexecuted by the arithmetic element, or may be configured as a controlcircuit that realizes functions of each unit. Each of the communicationunits from the first CAN communication unit 27 to the fifth CANcommunication unit 502 are configured by appropriately combining acommunication unit for connection with the CAN 90, a communicationinterface, and driver software.

FIG. 3 illustrates a flow chart of a flow of virtual image displayprocessing by the HUD 1.

When a main power of the HUD 1 is turned on (step S01/Yes), the camera70 starts capturing an image (step S10), the LiDAR 60 starts roadsurface measurement (step S20), the GPS receiver 80 receives GPS radiowaves, and the navigation device 800 starts acquisition of routeinformation (step S30). The HUD 1 is in a standby state until the mainpower of the HUD 1 is turned on (step S01/No).

The image processing device 700 reads a captured image from the camera70 and performs image recognition processing (step S11). Here, the imagerecognition processing is performed for detecting a subject reflected inthe captured image and determining whether the subject is a subjectwhich is a virtual image display target object of the HUD 1. In thefirst embodiment, it is assumed that the virtual image display targetobject is a route display object. Accordingly, the third ECU 701determines whether at least one of the front road surface, anintersection, a branch point, a junction point, and a corner isreflected on the captured image, and when at least one of them isreflected on the captured image, outputs the virtual image displaytarget object information to the CAN 90.

The road surface estimation device 600 acquires measurement point datafrom the LiDAR 60 and estimates the front road surface (step S21). Then,the road surface estimation device 600 outputs, to the HUD controller20, the road surface information of the front road surface 210 withrespect to the ground surface 200 on which the vehicle 2 contacts.

The navigation device 800 generates course information including thecurrent position and the traveling direction of the vehicle 2 based onthe GPS radio waves from the GPS receiver 80 and outputs the courseinformation to the HUD controller 20.

When determining that a virtual image display target object exists basedon the virtual image display target object information (step S40/Yes),the road surface estimation device 600 calculates a road surface angleθ_(r) of the front road surface 210, which has been estimated by theroad surface estimation device 600, with respect to the ground surface200 (step S41). The road surface angle θ_(r) (see FIG. 4) has the samemeaning as the slope of the front road surface 210 with respect to theground surface 200.

The HUD controller 20 calculates the height of the display object in thereal space based on the road surface angle θ_(r) obtained in step S41(step S42). Furthermore, the HUD controller 20 calculates a displayobject rotation angle for displaying the virtual image 101 of thedisplay object (see FIG. 1) along the front road surface 210 having theslope (step S43). Details of the display object rotation angle will bedescribed later.

Still further, the HUD controller 20 calculates a virtual image planeposition P′″_(oi) (see FIG. 7) on the virtual image plane 100, which isa position where the virtual image 101 of the display object is to bedisplayed, based on the slope obtained in step S41 (step S44). Detailedprocessing contents of each step will be described later.

Thereafter, the HUD controller 20 displays the display object at aposition on a display surface of the display element 33 corresponding tothe virtual image surface position P′″_(oi), and emits the image light Lso as to display the display object as the virtual image 101(corresponding to AR display) (step S45). Since the virtual image 101 isdisplayed on an intersection point between the virtual image plane 100and a line of sight at which the driver 5 views the virtual imagedisplay target object, the display object can be superimposed on orbrought close to the virtual image display target object (for example, aroad surface or an obstacle) by performing AR display.

When the HUD 1 is not turned off (step S46/No), the processing returnsto steps S10, S20, and S30 and is continued. On the other hand, when theHUD 1 is turned off (step S46/Yes), the processing is terminated.

(Step S21: Front Road Surface Estimation Processing)

FIG. 4A illustrates an outline of the estimation processing of the frontroad surface 210 (y-z plane), and FIG. 4B illustrates an outline of theestimation processing of the front road surface 210 (x-y plane). Each ofthe x-axis, the y-axis, and the z-axis in FIG. 4A and FIG. 4B is an axiswhich forms the three-axis rectangular coordinate system in the realspace as described above. The origin of the three-axis rectangularcoordinate system is positioned on the road surface which is forwardmostfrom the vehicle 2, in other words, on a plane extending the groundsurface 200 at infinity.

It is assumed that the measurement point of the front road surface 210measured by the LiDAR 60 on the three-dimensional real space is P_(i)(i=1, 2, 3, . . . ). The coordinates of the measurement point P_(i) isformed by (x_(i), y_(i), z_(i)). FIG. 4A and FIG. 4B illustrate threemeasurement points P₁, P₂, P₃. When a plane A including each measurementpoint P₁, P₂, P₃ is “ax+by +cz=d”, a, b, c, and d can be calculated bythe mathematical formula (1) below.

$\begin{matrix}\left. \begin{matrix}{{P_{12} \times P_{13}} = \left( {a,b,c} \right)^{T}} \\{d = {{ax}_{1} + {by}_{1} + {cz_{1}}}}\end{matrix} \right\} & (1)\end{matrix}$

Where, P₁₂ is a vector with P₁ and P₂, and P₁₃ is a vector with P₃.Multiplication×expresses an outer product operation.

(Step S41: Calculation Processing of Road Surface Angle θ_(r) of FrontRoad Surface 210 with Respect to Ground Surface 200)

Since the z-axis is included in the ground surface 200, the slope of thefront road surface 210 with respect to the ground surface 200, in otherwords, the road surface angle θ_(r) can be expressed by the slope withrespect to the z-axis in FIG. 4. Since the normal vector of the plane Aincluding the front road surface 210 is “n=(a, b, c)^(T)”, the roadsurface angle θ_(r) of the front road surface 210 can be calculated bythe mathematical formula (2) below.

$\begin{matrix}\left. \begin{matrix}{{\cos\;\theta} = \frac{c}{n}} \\{{n} = \sqrt{a^{2} + b^{2} + c^{2}}} \\{\theta = {\cos^{- 1}\frac{c}{n}}} \\{\theta_{r} = {\frac{\pi}{2} - \theta}}\end{matrix} \right\} & (2)\end{matrix}$

(Step S42: Height of Display Object in Real Space)

FIG. 5 illustrates a three-dimensional position in the real space inwhich the display object is to be drawn (y-z plane). In FIG. 5, thedisplay object 120 is not along the front road surface 210 and isdisplayed in the same shape as the ground surface 200 withoutconsidering the slope. The display object height can be obtained byusing the road surface angle θ_(r), meanwhile, the processing forobtaining the display object height by using a plane estimation formulawill be described below since the case of obtaining the display objectheight by using the plane estimation formula is easier to be understood.

In FIG. 5, it is assumed that a position in the depth direction of thethree-dimensional position P_(o) in the real space in which the displayobject is to be drawn is z_(o), a position in the horizontal directionthereof is x_(o), and a position in the height direction thereof isy_(o). The position y_(o) in the height direction can be calculatedbased on the estimation formula for the plane A by using themathematical formula (3) below.

$\begin{matrix}{y_{o} = \frac{d - {ax_{o}} - {cz_{o}}}{b}} & (3)\end{matrix}$

Where, the three-dimensional position in the real space in which thedisplay object is to be drawn is P_(o)=(x_(o), y_(o), z_(o)).

(Step S43: Calculation of Rotation Angle of Display Object)

FIG. 6A illustrates a shape of a display object parallel to the groundsurface 200 (left in FIG. 6A; basic shape) and a shape of the displayobject obtained by rotating the basic shape thereof in accordance with aroad surface angle (right in FIG. 6A; rotation shape). FIG. 6Billustrates a state where the display object 120 is displayed along thefront road surface 210 (y-z plane).

As illustrated in FIG. 6A, in order to display the pointsP_(oi)=(x_(oi), y_(oi), z_(oi))^(T) (i=1, 2, 3, . . . ) on the displayobject 120 (basic shape), which is displayed on the ground surface 200,along the front road surface 210 inclined at the road surface angleθr_(i) the points P_(oi) are rotated based on the road surface angleθ_(r) by using the mathematical formula (4) below so as to obtain thecoordinates of the points P′_(oi). The points P′_(oi) are points on adisplay object 120 a after the rotation. In the following, it is assumedthat the center of gravity of the virtual image of the display object120 is positioned at the origin.

$\begin{matrix}{p_{oi}^{\prime} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{r}} & {\sin\;\theta_{r}} \\0 & {{- {s{in}}}\;\theta_{r}} & {\cos\;\theta_{r}}\end{pmatrix}\begin{pmatrix}x_{oi} \\y_{oi} \\z_{oi}\end{pmatrix}}} & (4)\end{matrix}$

By performing the steps above, the display object 120 is rotated inaccordance with the road surface angle θ_(r) of the front road surface210 with respect to the ground surface 200 by using the measurementpoint data from the LiDAR 60 mounted on the vehicle 2. As illustrated inFIG. 6B, the display object 120 a after the rotation is moved to theposition P″_(oi) on the front road surface 210 at which the displayobject 120 a is to be displayed. In this connection, the positionP″_(oi) has the coordinates expressed by the three-axis rectangularcoordinate system in the real space. The coordinates of P″_(oi) can beexpressed by the mathematical formula (5) below.

p″ _(oi) =p′ _(oi) +p ₀  (5)

(Step S44: Calculate Virtual Image Plane Position of Display Object)

FIG. 7 illustrates a position on the virtual image plane 100 fordisplaying the virtual image (virtual image plane position P′″_(oi)) inorder to display the virtual image of the display object 120 a after therotation along the front road surface 210 (y-z plane). When the virtualimage plane 100 in the real space is “ex+fy+gz=h” and the viewpoint isP_(E), the position P′″_(oi) of the display object on the virtual imageplane can be obtained by the mathematical formula (6) below. It ispreferable that the coordinates of the viewpoint P_(E) on the three-axisrectangular coordinate system in the real space are the coordinates ofthe actual viewpoint of the driver 5 of the vehicle 2, meanwhile, in thepresent embodiment, a fixed value (for example, may be a design valuesuch as the center coordinates of the eyelips) is used for convenienceof explanation, and it is assumed that the three-axis rectangularcoordinates of the viewpoint P_(E) in the real space is provided inadvance. In this connection, as another mode, it may be configured toconnect a viewpoint detection device for detecting the viewpoint of thedriver 5 to the HUD 1 so as to utilize the coordinates of the viewpointP_(E) defined by the three-axis rectangular coordinate system detectedby the viewpoint detection device.

$\begin{matrix}{p_{oi}^{\prime\prime\prime} = {p_{E} + {\frac{h - \left( {{ex_{E}} + {fy_{E}} + {gz_{E}}} \right)}{\left( {{e\left( {x_{oi}^{''} - x_{E}} \right)} + {f\left( {y_{oi}^{''} - y_{E}} \right)} + {g\left( {z_{oi}^{''} - z_{E}} \right)}} \right)}\left( {p_{oi}^{''} - p_{E}} \right)}}} & (6)\end{matrix}$

(Step S45: Virtual Image Display)

FIG. 8 illustrates positional relationship between the virtual imageplane position and a position on a display surface 33 a of the displayelement 33. The virtual image plane-and-display surface linkage data,which will be described later, is the relationship illustrated in FIG.8, that is, the data in which the x-z coordinates on the virtual imageplane and the s-t coordinates within the display surface are linked witheach other.

The light emitted from the light source 31 is transmitted through thedisplay surface 33 a of the display element 33, whereby the image lightL including the virtual image is generated. Accordingly, the area of thevirtual image plane 100 is expanded as the image light L is diffused,and the display position within the virtual image plane 100 (virtualimage plane position P′″_(oi)) and the corresponding display position onthe display surface 33 a of the display element 33 are uniquelydetermined.

Accordingly, the display object is displayed on the display surface 33 aof the display element 33, and the display position P′″_(oi) at thattime is expressed by a two-axis rectangular coordinate system (s-tcoordinate system) on the display surface 33 a. In addition, thecoordinates of the virtual image plane position P′″_(oi) at which thevirtual image of the display object displayed on the display positionP″″_(oi) is displayed are expressed by the x-y coordinates in thethree-axis rectangular coordinate system in the real space. Then, thevirtual image plane-and-display surface linkage data in which the s-tcoordinates of the display position P″″_(oi) and the x-y coordinates ofthe virtual image plane position P′″_(oi) are linked with each other isgenerated.

The virtual image plane-and-display surface linkage data is stored in acalibration data storage unit 22 a configured by a partial region of thefirst nonvolatile memory 22.

When obtaining the virtual image plane position P′″_(oi) based on themathematical formula (6), the first ECU 21 refers to the virtual imageplane-and-display surface linkage data to convert the virtual imageplane position P′″_(oi) into the display position P″″_(oi) on thedisplay surface 33 a. Thereafter, the first ECU 21 outputs the displayobject 120 a after the rotation and the display position P″″_(oi) on thedisplay element 33 to the distortion correction unit 25. The distortioncorrection unit 25 outputs the display object 120 a after the rotationand correction and the display position P″″_(oi) on the display surface33 a to the display element control unit 26. The display element controlunit 26 drives the display element 33 so as to display the displayobject 120 a after the rotation on the display position P″″_(oi) of thedisplay surface 33 a.

Then, the first ECU 21 outputs an instruction for lighting the lightsource 31 to the light source adjustment unit 24, and the light sourceadjustment unit 24 turns on the light source 31. Accordingly, the lightsource 31 outputs an emission light, and the image light L including thedisplay object displayed on the display position P″″_(oi) of the displaysurface 33 a is emitted from the HUD 1. By the image light L, thedisplay object 120 a after the rotation is displayed on the positionP′″_(oi) of the virtual image plane 100 as a virtual image. The virtualimage plane-and-display surface linkage data in FIG. 8 shows an exampleof data when the concave mirror 41 is rotated by a certain angle, forexample, φ₁. When the concave mirror angle φ₁ is changed, even if theposition P′″_(oi) of the virtual image plane 100 remains the same, thedisplay position of the display object on the display surface 33 acorresponding thereto is changed. For example, in the case of theconcave mirror angle φ₁, the position P′″_(oi) of the virtual imageplane 100 is expressed as (s_(a1), t_(a1)) in the s-t coordinate systemof P″″_(oi) while in the case of the concave mirror angle φ₂, it isexpressed as (s_(a2), t_(a2)). Accordingly, a plurality of pieces ofvirtual image plane-and-display surface linkage data_(i) correspondingto the concave mirror angle φ_(i) is stored in advance in thecalibration data storage unit 22 a, and the first ECU 21 reads out thevirtual image plane-and-display surface linkage data_(i) by referring toa signal for controlling the concave mirror angle φ_(i) outputted to theconcave mirror control unit 28, and calculates the display position ofthe display object 120 a on the display surface 33 a.

With reference to FIGS. 9 to 11, an effect of the present embodimentwill be described by comparing the present embodiment with the priorart.

FIG. 9 illustrates an example in which AR display is performed for thefront road surface without change in slope (prior art). In FIG. 9,arrows indicating the traveling direction are displayed along the frontroad surface that is on the same plane as that of the ground surface200.

FIG. 10 illustrates an example in which the AR display is performed forthe front road surface 210 having slope, without considering the slope.The front road surface 210 has upward slope with respect to the groundsurface 200, and thus the front road surface 210 is present at a higherposition than the ground surface 200 in the real space. When the ARdisplay is performed for the front road surface 210 without consideringthe slope, each display object of the arrows appears as if it pierces orpenetrates the front road surface 210.

FIG. 11 illustrates an example in which the AR display is performed forthe front road surface having change in slope, with considering theslope (corresponding to the present embodiment). As illustrated in FIG.11, by changing the display height of the display object withconsidering the slope (road surface angle θ_(r)) with respect to thefront road surface 210 having change in slope and rotating the displayobject for display, it is possible to display each display object of thearrows along the front road surface 210 without being appeared to pierceor penetrate the front road surface 210.

According to the present embodiment, the output from the LiDAR 60 isused to obtain the road surface angle θ_(r) of the front road surface210, and the basic shape of the display object is rotated based on theroad surface angle θ_(r). Furthermore, the height of the display objectfor displaying the display object along the slope of the front roadsurface 210 is obtained, and the virtual image plane position of thedisplay object for realizing the display above is calculated. Bydisplaying the display object after the rotation at the display positionP″″_(oi) on the display surface 33 a corresponding to the virtual imageplane position P″″_(oi) and emitting the image light L, even if thefront road surface 210 has the slope with respect to the ground surface200, it is possible to display the display object along the front roadsurface 210 as compared with the prior art.

Furthermore, according to the present embodiment, the road surface angleθ_(r) of the front road surface 210 with respect to the ground surface200 is measured by using the output of the LiDAR 60 mounted on thevehicle 2, and the display position and the rotational angle of thedisplay object are changed based on the measured road surface angleθ_(r), which may be realized by replacing the output of the LiDAR 60with road surface shape information of a high-precision map. However,depending on the update frequency of the map, there is a possibilitythat a road state and the map information do not match with each other,and if the road state or road slope changes from the time when the mapinformation is created to the current time, there is a possibility thatcorrect road shape information of the current state cannot be acquired.On the other hand, according to the present embodiment, the road stateand the road slope are followed in real time by successively acquiringroad shape information so as to perform the AR display, and therefore,it is possible to realize the AR display more suited to the real scenewhile avoiding the AR display without reality, in which for example, thevirtual image penetrating the front road surface 210 is displayed.

Second Embodiment

The second embodiment includes the technical feature in which thevirtual image display target object is an obstacle such as pedestriansand vehicles. Since the three-dimensional coordinates of the detectedobstacle can be estimated by using the position and the road surfaceinformation on the camera projection plane, when the image processingdevice 700 detects the obstacle such as pedestrians or vehicles, it ispossible to perform the AR display with high accuracy.

The technical feature of the second embodiment can be found in thefollowing three processes as compared with the first embodiment.

(First Process)

Link the road surface information acquired by the LiDAR 60 with theimage captured by the camera 70.

(Second Process)

Obtain measurement points on which the obstacle (which is an example ofthe virtual image display target object) detected on the captured imageexists.

(Third Process)

Display the virtual image of the display object to be added to theobstacle along the plane including the measurement points in thevicinity.

(Regarding First Process)

The first process is performed before start of the processes in FIG. 3.In the first process, camera-and-real space linkage data, in whichcoordinates of a camera coordinate system and coordinates of thethree-axis rectangular coordinate system in the real space are linkedwith each other, is generated by installing a subject having known sizeat a known position to which the distance from the vehicle 2 is known,measuring the subject by each of the camera 70 and the LiDAR 60, andlinking the subject in the captured image with positions of measurementpoints on the subject detected by the LiDAR 60.

The subject in the captured image can be expressed by thetwo-dimensional coordinates of the camera coordinate system.Accordingly, when the LiDAR 60 measures the measurement points on thesame subject, the coordinates of the three-axis rectangular coordinatesystem in the real space of each measurement point can be obtained,whereby the camera-and-real space linkage data in which the coordinatesof the three-axis rectangular coordinate system and the two-dimensionalcoordinates of the camera coordinate system are linked with each othercan be obtained. Thus, the two-dimensional coordinates of the subject inthe captured image, which is expressed by the camera coordinate system,and the three-axis rectangular coordinates of the subject detected bythe LiDAR 60 are identified by using the camera-and-real space linkagedata. Accordingly, it is possible to calculate the three-dimensionalcoordinates of the subject in the captured image.

The camera-and-real space linkage data is stored in the calibration datastorage unit 22 a, and is used in processing of determining the virtualimage plane position of the display object, etc.

Specifically, in the camera-and-real space linkage data, linkage betweenlocations of pixels of the captured image by the camera 70 and locationsof the measurement point data (group of points p_(l) consisting of x, y,and z components) acquired by the LiDAR 60 is specified. Generally,since the installation position of the camera 70 is different from theinstallation position of the LiDAR 60, the camera coordinate system witha camera position as the origin and a LiDAR coordinate system with aLiDAR position as the origin are taken into consideration. FIG. 12A andFIG. 12B illustrate an outline of the first process. FIG. 12Aillustrates a state where an obstacle 650 is viewed in the LiDARcoordinate system. FIG. 12B illustrates a state where the same obstacle650 as that in FIG. 12A is viewed in the camera coordinate system. InFIG. 12A and FIG. 12B, the coordinates are provided with subscripts cand 1, respectively.

FIG. 12A assumes a three-dimensional position P_(li) (i=1, 2, 3) of theroad surface measured by the LiDAR 60. FIG. 12B illustrates athree-dimensional position P_(ci) (i=1, 2, 3) in the camera coordinatesystem.

It is assumed that the difference between the position of the LiDAR 60and the position of the camera 70 is expressed by a translation vectort, and the difference of attitude therebetween is expressed by arotation matrix R. The road surface position P_(ci) in the cameracoordinate system can be calculated by the mathematical formula (7)below.

P _(ci) =RP _(li) +t  (7)

Next, as illustrated in FIG. 13A and FIG. 13B, the road surface positionp_(ci) on the camera coordinates is converted into the position P_(pi)on the camera projection plane and the position P_(ii) on the image(FIG. 13B). When the focal length of the camera 70 is f, the positionP_(pi) on the camera projection plane can be calculated by themathematical formula (8) below.

$\begin{matrix}{p_{pi} = {f\frac{1}{z_{ci}}p_{ci}}} & (8)\end{matrix}$

Furthermore, when the width of an image sensor of the camera is w_(S)and the height thereof is h_(S), and when the width of the image isw_(I) and the height thereof is h_(I), the position P_(ii) on the imagecan be calculated by the mathematical formula (9) below.

$\begin{matrix}{p_{ii} = {{{f\begin{pmatrix}\frac{w_{I}}{w_{S}} & 0 \\0 & \frac{h_{I}}{h_{S}}\end{pmatrix}}p_{pi}} + \begin{pmatrix}\frac{w_{I}}{2} \\\frac{h_{I}}{2}\end{pmatrix}}} & (9)\end{matrix}$

(Regarding Second Process)

As illustrated in FIG. 13B, in the second embodiment, the first ECU 21confirms whether an obstacle P_(iv) exists among the road surfacepositions P_(i1), P_(i2), P_(i3) on the captured image in step S2. Whenall the mathematical formulas (10) to (12) are satisfied, the first ECU21 determines that the obstacle P_(iv) exists among the road surfacepositions P_(i1), P_(i2), P_(i3).

(P_i3−P_i1)×(P_i v−P_i1)<0  (10)

(P_i2−P_i3)×(P_i v−P_i3)<0  (11)

(P_i1−P_i2)×(P_i v−P_i2)<0  (12)

*Where, multiplication×expresses an outer product operation.

(Regarding Third Process)

When the second process above is satisfied, the first ECU 21 determinesthat the obstacle exists on the plane A including P₁₁, P₁₂, P₁₃ in step40.

Since an intersection point between the plane A and a straight lineconnecting the camera origin and the obstacle position P_(pv) on thecamera projected plane is the position P_(cv) of the obstacle in thereal space, the first ECU 21 calculates the position P_(cv)=(x_(cv),y_(cv), z_(cv)) of the obstacle in the real space obtained in the secondprocess by the mathematical formula (13). In the mathematical formula(13), P_(cv)=(x_(cv), y_(cv), z_(cv)) expresses the position of theobstacle on the camera projection plane.

$\begin{matrix}\left. \begin{matrix}{p_{pv} = {\begin{pmatrix}\frac{w_{S}}{{fw}_{I}} & 0 & 0 \\0 & \frac{h_{S}}{{fh}_{I}} & 0 \\0 & 0 & f\end{pmatrix}\begin{pmatrix}{x_{iv} - \frac{w_{I}}{2}} \\{y_{iv} - \frac{h_{I}}{2}} \\1\end{pmatrix}}} \\{p_{cv} = {tp}_{pv}} \\{t = \frac{d}{\left( {{ax_{p\nu}} + {by_{pv}} + {cz_{pv}}} \right)}}\end{matrix} \right\} & (13)\end{matrix}$

(Step S44: Virtual Image Plane Position of Obstacle)

FIG. 14 illustrates a virtual image plane position of the obstacle (y-zplane). The first ECU 21 of the HUD controller 20 obtains the positionof the obstacle on the virtual image plane in the same manner as stepS44 described above. Since the position P_(cv) is the obstacle positionwith the camera position as the origin, the position P_(cv) is convertedinto a position P_(v) in the coordinate system defined in the firstprocess by using the camera-and-real space linkage data obtained in thefirst process. When the difference of the position and orientationbetween the camera coordinates and the coordinate system defined in thefirst process is a translation vector t₂ and a rotation matrix R₂, theposition P_(v) is calculated by the mathematical formula (14) below.

P _(v) =R ₂ P _(cv) +t ₂  (14)

When the virtual image plane 100 in the real space is “ex+fy+gz=h” andthe viewpoint is P_(E), the position P′_(v) of the obstacle on thevirtual image plane 100 can be obtained by the mathematical formula (15)below.

$\begin{matrix}{p_{v}^{\prime} = {p_{E} + {\frac{h - \left( {{ex}_{E} + {fy_{E}} + {gz_{E}}} \right)}{\left( {{e\left( {x_{v} - x_{E}} \right)} + {f\left( {y_{v} - y_{E}} \right)} + {g\left( {z_{v} - z_{E}} \right)}} \right)}\left( {p_{v} - p_{E}} \right)}}} & (15)\end{matrix}$

According to the second embodiment, it is possible to perform the ARdisplay even for the display object which appears irregularly like anobstacle and cannot be defined by the map information.

An effect of the AR display according to the second embodiment will bedescribed by comparing the second embodiment with the prior art. FIG.15A illustrates an example in which the AR display is performed for anobstacle without considering the slope of the front road surface. FIG.15B illustrates an example in which the AR display is performed for theobstacle with considering the slope of the front road surface.

In the case of without considering the slope, as illustrated in FIG.15A, a display object 660 is displayed separately from the obstacle 650which is a forward vehicle. In contrast, according to the presentembodiment, as illustrated in FIG. 15B, the display object 660 can bedisplayed closely to the obstacle 650. In FIG. 15B, the display object660 is displayed closely to the obstacle 650, meanwhile, it is alsopossible to display the obstacle 650 by superimposing the display object660 thereon.

Third Embodiment

In the first embodiment, the height of the display object in the realspace is obtained based on the estimation formula of the front roadsurface which is expressed by the three-axis rectangular coordinatesystem. Meanwhile, the height of the display object in the real spacealso can be obtained by using the slope (road surface angle θ_(r)) ofthe front road surface 210 with respect to the x-z axis (ground surfaceon which tires contact) in the three-axis rectangular coordinate system.FIG. 16 illustrates processing for obtaining the height of the displayobject in the real space according to the third embodiment.

In FIG. 16, it is assumed that a mathematics formula expressing theplane of the front road surface 210 is “ax+by +cz=d”. The point of theposition x₀ (x₀, 0, z′) on the intersection line between the plane andthe ground surface 200 can be expressed based on “ax₀+0+cz′=d” by themathematical formula (16) below.

$\begin{matrix}{z^{\prime} = \frac{d - {ax}_{0}}{c}} & (16)\end{matrix}$

Next, the height y₀ of the display object in the real space is obtainedby the mathematical formula (17) below.

$\begin{matrix}\left. \begin{matrix}{\frac{y_{o}}{z_{o} - z^{\prime}} = {\tan\;\theta_{r}}} \\{y_{o} = {\left( {z_{o} - z^{\prime}} \right)\tan\theta_{r}}} \\{z^{\prime}\mspace{14mu}{is}\mspace{14mu}{substituted}} \\{y_{o} = {\left( {z_{o} - \frac{d - {ax_{o}}}{c}} \right)\tan\theta_{r}}}\end{matrix} \right\} & (17)\end{matrix}$

Fourth Embodiment

In the first to third embodiments, the image display device 30 using anLCD is used. Meanwhile, the image display device 300 using a MEMS (MicroElectro Mechanical Systems) may be used. FIG. 17 is a systemconfiguration diagram of the HUD according to the fourth embodiment.

An HUD 1 a illustrated in FIG. 17 is configured by the image displaydevice 300 using the MEMS. The MEMS includes a laser light source 301, ascanning mirror 302 that reflects a laser light, a scanning mirror driveunit (motor) 302 a configured to change an angle of a mirror surface ofthe scanning mirror 302, a diffusion plate 303 using a microlens array,and a relay optical system 304 configured to receive the image lightfrom the diffusion plate 303 and output the image light toward theconcave mirror 41. The relay optical system 304 is a member that isprovided in place of the lens unit 43 in FIG. 2.

The laser light irradiated from the laser light source 301 is reflectedby the scanning mirror 302 and reaches the diffusion plate 303. Thescanning mirror 302 irradiates the laser light to the diffusion plate303 with a reflection angle thereof being changed. On the diffusionplate 303, the laser light forms an image once so that the displayobject can be visually recognized. Accordingly, the diffusion plate 303corresponds to the display surface 33 a. The light directed from thediffusion plate 303 to the relay optical system 304 corresponds to theimage light L since it includes image information of the imaged displayobject.

An HUD controller 20 a provided in the HUD 1 a includes a scanningmirror control unit 26 a in place of the display element control unit 26of the HUD controller 20 according to the first embodiment. The lightsource adjustment unit 24 is connected to the laser light source 301 toperform blink control and light amount adjustment.

The scanning mirror control unit 26 a drives and controls the scanningmirror drive unit 302 a by rotating the scanning mirror 302 to changethe orientation of the mirror surface. With this configuration, theposition on the diffusion plate 303, at which the display object isdisplayed, is changed.

Furthermore, the HUD controller 20 a includes an input I/F 27 a. Each ofthe LiDAR 60, the camera 70, and the navigation device 800 is connectedto the input I/F 27 a. The first ECU 21 acquires the respective outputsfrom the LiDAR 60, the camera70, and the navigation device 800 throughthe input I/F 27 a, and performs image processing such as road surfaceestimation and obstacle detection, thereby realizing the AR displaywhich is the same as that of the first embodiment.

Still further, the HUD controller 20 a may be configured to acquire thevehicle information such as the traveling speed from a speed sensor 950via the CAN 90 so as to provide the vehicle information by the ARdisplay.

According to the present embodiment, the image display device 300 usingthe MEMS also can perform the AR display in accordance with the slope ofthe front road surface. Furthermore, even in the case of a vehicle thatdoes not mount the automatic operation system 900 which is mounted inthe first embodiment, by attaching various sensors to the HUD 1 a, theAR display can be performed in accordance with the slope of the frontroad surface.

Each of the above-described embodiments does not limit the presentinvention, and various modifications within a scope that does not departfrom the concept of the present invention belong to the technical scopeof the present invention. For example, each mathematical formula usedfor explaining each processing is only one embodiment of the processing,and other mathematical formulas that produce calculation resultsnecessary for the processing may be applied.

The road surface detection sensor is not limited to the LiDAR 60, and amillimeter-wave radar or a stereo camera can be used as long as it candetect the distance and position to a measurement point on the frontroad surface 210 (position in the lateral direction of the vehicle 2).

Furthermore, in place of the image display device 30 using the LCDaccording to the first to third embodiments, the image display device300 using the MEMS may be used. Still further, in place of the imagedisplay device 300 using the MEMS according to the fourth embodiment,the image display device 30 using the LCD may be used.

REFERENCE SIGNS LIST

-   1: HUD-   20: HUD controller (main controller)-   60: LiDAR-   70: camera-   80: GPS receiver-   100: virtual image plane-   101: virtual image-   200: ground surface-   210: front road surface

1. A head-up display for irradiating an image light including a displayobject toward a projection target member so as to display the displayobject as a virtual image, the head-up display comprising: an imagedisplay device including a light source and a display surface, the imagedisplay device being configured to output the image light generated by alight which has been emitted from the light source and transmittedthrough the display object displayed on the display surface; a virtualimage optical system configured to enlarge and project the image light;and a main controller connected to the image display device, wherein themain controller is configured to: acquire road surface information of afront road surface based on a plurality of measurement points on thefront road surface, the plurality of measurement points being detectedby a road surface detection sensor mounted on a vehicle with the head-updisplay and being positioned forward of a traveling direction of thevehicle; calculate, by using the road surface information, a virtualimage plane position on a virtual image plane where the virtual image ofthe display object is displayed, the virtual image plane position beinga display position of the virtual image for displaying the virtual imagealong the front road surface; and calculate a display position of thedisplay surface corresponding to the virtual image plane position so asto output, to the image display device, a control signal for displayingthe display object on the display position of the display surface. 2.The head-up display according to claim 1, wherein the main controller isfurther configured to acquire the road surface information of the frontroad surface based on three-dimensional coordinates of each of thedetected plurality of measurement points, and the three-dimensionalcoordinates are defined by a three-axis rectangular coordinate system ina real space.
 3. The head-up display according to claim 1, wherein themain controller is further configured to calculate the display positionof the display surface corresponding to the virtual image plane positionbased on the virtual image plane position of the virtual image in thereal space by referring to virtual image plane-and-display surfacelinkage data in which coordinates on the virtual image plane andcoordinates on the display surface are linked with each other.
 4. Thehead-up display according to claim 1, wherein the main controller isfurther configured to calculate, as the virtual image plane position ofthe virtual image, a position of a point at which a line of sightconnecting a viewpoint of a driver who visually recognizes the virtualimage with the front road surface intersects the virtual image plane. 5.The head-up display according to claim 2, wherein the three-axisrectangular coordinate system includes an x-z axis rectangularcoordinate system included in a ground surface on which a traveling bodymounted on the vehicle contacts and a y-axis perpendicular to the x-zaxis rectangular coordinate system, and the main controller is furtherconfigured to calculate, as the virtual image plane position, heightfrom the ground surface for displaying the virtual image on the virtualimage plane.
 6. The head-up display according to claim 5, wherein themain controller is further configured to acquire an: for a planeincluding the plurality of measurement points as the road surfaceinformation, and calculate the virtual image plane position based on theestimation formula.
 7. The head-up display according to claim 5, whereinthe main controller is further configured to: further acquire orcalculate, as the road surface information, a road surface angle whichis an angle formed by a plane including the plurality of measurementpoints with respect to the ground surface; and set a shape of thedisplay object at a time of being displayed on the ground surface as abasic shape so as to display the display object after rotation, which isobtained by rotating the basic shape based on the road surface angle, atthe display position for the display object on the image display device.8. The head-up display according to claim 5, wherein the main controlleris further configured to: acquire, as the road surface information, aroad surface angle which is an angle formed by a plane including theplurality of measurement points with respect to the ground surface;calculate the virtual image plane position of the virtual image by usingthe road surface angle; and set a shape of the display object at a timeof being displayed on the ground surface as a basic shape so as todisplay the display object after rotation, which is obtained by rotatingthe basic shape based on the road surface angle, at the display positionfor the display object on the image display device.
 9. The head-updisplay according to claim 1, wherein the main controller is furtherconfigured to: acquire positional information of a virtual image displaytarget object in a captured image of the front road surface which hasbeen captured by a camera mounted on the vehicle; refer tocamera-and-real space linkage data in which coordinates of a three-axisrectangular coordinate system in a real space and two-dimensionalcoordinates of a camera coordinate system are linked with each other soas to convert the positional information of the virtual image displaytarget object into the coordinates of the three-axis rectangularcoordinate system; refer to the positional information of the virtualimage display target object after conversion and the road surfaceinformation so as to calculate a plane including the plurality ofmeasurement points near the virtual image display target object, andcalculate the virtual image plane position for displaying a virtualimage of a display object to be added to the virtual image displaytarget object along the plane; refer to virtual image plane-and-displaysurface linkage data, in which coordinates on the virtual image planeand coordinates on the display surface are linked with each other, so asto convert the virtual image plane position of the display object to beadded to the virtual image display target object into the displayposition of the display surface; and output, to the image displaydevice, a control signal for displaying the display object to be addedto the virtual image display target object on the display position ofthe display surface.
 10. The head-up display according to claim 1,wherein the image display device is a liquid crystal display including:a light source; a display element; and an illumination optical systemdisposed between the light source and the display element, which isconfigured to guide a light emitted from the light source to the displayelement.
 11. The head-up display according to claim 1, wherein the imagedisplay device is a microelectromechanical system including: a laserlight source; a scanning mirror that reflects a laser light emitted fromthe laser light source; a scanning mirror drive unit configured tochange an orientation of a mirror surface of the scanning mirror; adiffusion plate that forms an image by the laser light reflected by thescanning mirror; and a relay optical system on which an image lightincluding image information of the display object, which has been formedon the diffusion plate, is made incident.